For photography at events, theme parks and public places, it is difficult to process personalized souvenirs quickly. It would be even better if eye-catching 3D souvenirs could be personalized and produced for visitors, but converting images or video to 3D is difficult in real-time. Adding service personnel doesn't solve the problem, if a lot of manual processing is required, because of the high labor costs. Plus, even with more people involved, the problem of creating a realistic 3D model from a two-dimensional photo is no easier.
Automatic segmentation of foreground objects in digital video is often described as an unsolved scientific problem. Biological visual systems make it seem easy. Adding supplementary data like laser range-finders or infrared patterns can improve depth perception in computer vision. But that adds additional hardware costs and complexity. For image processing with an ordinary photo or movie clip, segmenting objects depth-wise is sometimes called the Holy Grail of content conversion.
It is possible to pre-render a movie in 3D to avoid the need for real-time automatic segmentation. The movie studios have been making 3D animated cartoon movies that are not photorealistic based on 3D modeling. This is also the state-of-the-art for modern 3D computer games. However, the 3D is still locked inside a flat display screen in most cases. There is a growing resurgence of movie theaters showing 3D movies with stereoscopic glasses based on polarized or shutter lenses. It would be nice if customers could take home personalized souvenirs such as 3D pictures and video. However, until now, only big-budget movies have been able to create 3D. And even then, it is pre-processed over months or years of work.
A further problem is the need for glasses-free 3D. 3D glasses have been criticized for poor viewing. Red-blue anaglyphs cause ghosting and headaches. Polarized and shutter glasses are inconvenient and potentially expensive in large quantities for movie theaters. There are also binocular video visors, although these are not transparent and prevent interaction with others and observation of the surrounding environment.
A further problem is that sometimes it is not possible to do a lenticular print to show a 3D image or video. Laminating a lenticular lens onto an interlaced print is too time-consuming and difficult for most consumers. They would like to have instant 3D photos, but there is no obvious way to do this themselves. A mobile phone can display the imagery, but it too may not have a lenticular overlay available. It is also difficult to put an overlay on a mobile device, and to hold it in the precise location and orientation needed. A further limitation is that if the lenticular lenses are kept vertical, they usually cause Moiré Effects, magnifying the red, green or blue sub-pixels on a liquid crystal diode (LCD), light-emitting diode (LED), cathode ray tube (CRT) or other type of display screen. Yet the more they are tilted away from vertical, the less stereoscopic pop-out there is. A further problem is that most lenticular lens available is designed for printing, although the ink location has a different focal length than the pixels within a display screen.
Alternatively, a camera could be used to track the user's movements, and convert these into navigation input from the mouse or arrow keys on a computer. In this way, as the person moved, the perspective changes on-screen in a 3D scene, computer game or virtual tour. On an ordinary tablet or smartphone, you could move back and forth and see around foreground objects, like in a hologram. Unfortunately, room lighting varies dramatically for different users, which causes large variations in performance. A method is needed for when no optical overlay has been obtained for autostereoscopic viewing of a mobile display, that is more reliable than tracking from the built-in camera for viewing the “look-around effect” on each side of foreground objects on-screen.
A machine that could solve these problems would be very useful.